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To detect human body and remove noises from complex background, illumination variations and 
objects, the infrared thermal imaging was applied to collect gait video and an infrared thermal gait 
database was established in this paper. Multi-variables gait feature was extracted according to a novel 
method combining integral model and simplified model. Also the wavelet transform, invariant 
moments and skeleton theory were used to extract gait features. The support vector machine was 
employed to classify gaits. This proposed method was applied to the infrared gait database and 
achieved 78%-91% for the probability of correct recognition. The recognition rates were insensitive for 
the items of holding ball and loading package. However, there was significant influence for the item of 
wearing heavy coat. The infrared thermal imaging was potential for better description of human body 
moving within image sequences. 

© 2010 Elsevier Ltd. All rights reserved. 


1. Introduction 

Biometric recognition refers to an automatic recognition of 
individuals based on feature derived from their anatomical and/or 
behavioral characteristics [1,2]. We can obtain the anatomical 
characteristics of the human beings by measuring their finger¬ 
print, face, iris, hand shape and so on, and extract the behavioral 
characteristics from their behaviors, such as gait, voice and 
keystroke. Unlike conventional recognition techniques of pass¬ 
words or ID cards, biometric recognition technique is safer and 
more reliable because it is almost impossible to be disguised, 
shared, and misplaced [3]. This makes biometric recognition 
much more difficult to be abused than traditional methods of 
identification. 

Gait recognition, as one of new fields in biometric recognition, 
has unique capability to recognize people at a distance when 
other biometrics might not be perceivable. Furthermore, it is 
difficult to disguise gait without hampering progress, which is of 
particular interest in scene of crime analysis [4]. Gait can be 
detected and measured at low resolution, and therefore it can be 
used in situations where face or iris information is not available in 
high enough resolution for recognition [5]. However, there are a 
lot of challenges in gait recognition, such as foreground 
segmentation, clothing of the subject, walking speed change, 
carrying objects. In general, sensors of video collection include 
digital video (DV), professional camera and surveillance camera in 
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gait recognition. It is difficult for these sensors to remove noises 
from complex background, illumination variation and carrying 
objects. Fig. 1 illustrates some influencing factors. 

In order to improve the detection accuracy, we applied the 
technique of Infrared (IR) thermal imaging to recognize gaits. 
Human body is a natural emitter of infrared ray. Usually, the 
temperature of human body is different from that of background. 
This leads to the different energy distribution and gray-scale 
difference between background and human body in thermal 
image. Furthermore, if we use the technique of infrared thermal 
imaging, the environment will give little influence on the body 
detection. Especially, targets can be watched at night by using 
infrared thermal camera. Fig. 2 gives the comparison between 
conventional image and infrared image when subject walks with 
a package. The conventional images were taken outdoors in the 
afternoon. The outdoor lighting was very stable. Otherwise the 
complex environment would make the silhouette extraction 
more difficult. The package changes human body silhouette in 
the conventional image as shown in Fig. 2(b) and the impact from 
package is difficult to remove. However, Fig. 2(c) reveals that the 
package is invisible in infrared image because of different 
temperature between human body and the package. The clean 
silhouette can be generated for the infrared image as illustrated in 
Fig. 2(d). The infrared pattern facilitates silhouette extraction. So 
we collected infrared gait video with FLIR A40M thermal camera 
in our experiments. Daoliang Tan [6] used infrared camera to 
collect gait video at night. This technique is difficult to remove the 
influence from noises such as carrying object and illumination 
variations. Bhanu and Han [7,8] used FLIR SC2000 long-wave 
infrared thermal camera to deal with human activity recognition. 
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Each subject was asked to walk slowly, walk fast and run forward 
and backward. There were five subjects to be tested in their 
experiments. However, they did not analyze the situation for 
carrying objects. For this reason, we used the IR thermal camera to 
collect gait video and established a database of Infrared Thermal 
Gait with 23 subjects. Each subject was asked to walk with four 
conditions: walking in a natural manner, holding ball, loading 
package and walking with heavy coat. 

In addition, we used a novel method that the integral model was 
combined with the simplified model to obtain multi-variables gait 
features. The integral model is based on body silhouettes. By using 
this integral model, we can identify persons by analyzing motion 
information of silhouettes, such as walking speed, area change of 
silhouettes. The simplified model is based on the anatomic 
principles to simplify human body. But there is some limitation to 
extract gait feature by using single model (integral model or 
simplified model). The integral model based on silhouettes is 
excellent to recognize subjects fast and directly. But there is an 



indirect connection between the silhouette and the gait. Also 
complex background can influence the silhouette extraction. So 
the gait feature based on integral model is easy to be disturbed by 
background [9-11]. If we only use the simplified model to extract 
gait feature, some information (such as body width) will be lost after 
simplifying the human body. On the side, the gait feature generally 
includes behavioral characteristic or anatomical characteristic. How 
can we obtain the effective features including not only behavioral 
characteristic, but also anatomical characteristic? For this reason, 
this paper gives a novel method that integral model is combined 
with simplified model to obtain multi-variables information. 

At the same time, the wavelet transform (WT) and invariant 
moments were used to extract gait parameters. Also the grey- 
level gait energy image (GEI) was simplified to obtain body 
skeleton and to extract gait parameters. The GEI [12-15] reflects 
major shapes of silhouettes and their changes over gait cycle. So 
we extracted human silhouettes from the infrared video and 
obtained the GEI in this paper. These gait parameters were fitted 
together and presented to support vector machine for classifica¬ 
tion. Fig. 3 illustrates the technical flow chart. 


2. Methods 

2.1. Infrared video collection 

In our experiments, the infrared thermal camera A40M was 
used to collect gait video. This thermal camera is a tool for 
infrared detection technology. The kernel of camera is the infrared 



Fig. 2. Comparison of IR and conventional images: (a) conventional image, (b) silhouette in conventional image, (c) IR image and (d) silhouette in IR image. 
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Fig. 3. Technical flow chart. 


detector which can translate the infrared ray into electrical signal. 
The signal processing system can generate infrared image. In this 
way, the infrared gait video can be collected by using this thermal 
camera. Detection distance can take some impact on the quality of 
infrared image. The image of whole body cannot be obtained in 
short-distance between camera and human body. Also the long¬ 
distance will influence the image quality. In a set of experiments, 
thermal camera was about 6 m away from subjects and we 
established an infrared thermal gait database (ITGD). The 
temperature sensitivity of A40M is 0.08 °C for 30 °C. Frame 
frequency is 50/60 HZ and spatial resolution is 1.3mrad. Gait 
video was collected indoors and each subject was asked to 
walk with four conditions (normal walking, holding ball, loading 
package, wearing heavy coat). The image size is 320 x 240 pixels. 
Fig. 4 gives four images with four conditions. For the recognition 
performance, there are some important factors including complex 
background, carrying objects, heavy coat. So we designed 
these four situations and set static background indoor. However, 
it is obvious that more factors need to be taken into account 
in our dataset. This will be our future work for outdoor 
experiments. 


2.2. Wavelet decomposition of GEI 

(1) Gait Energy Image : The gait energy image (GEI) could be 
obtained by adding normalized silhouettes together. As shown in 


Fig. 4, the gray-value of body area is different from that of 
background because of the different temperature between human 
body and surroundings. So we can set a global threshold to extract 
body silhouette. The formula follows as 

f 1 |P;(X,y)| > T 

/i(X ' y) =(o |P i( x,y)| < t (1) 

In Eq. (1), Pi is gray-value in point (x,y) and t is the global 
threshold. Generally, the gray-value of surroundings is 30 below. 
The body’s gray-value is higher than 80. In this paper, we fixed 70 
as the global threshold. Thinking of the possibility for the hole, 
clearance and other noises in silhouettes, the binary image was 
processed by using morphological erosion and dilation. The 
formula follows as 

m 

£(x,y) = AND[f(x+i,y+j)&T(i,})] (2) 

l,] = 0 

m 

D(x,y) = . QRJf (x+i,y +j)&T(i,j)] (3) 

Eq. (2) is the erosion and Eq. (3) presents the dilation. Here, 
/(x,y) expresses the binary image and T(i,j) is structure element. 
The silhouettes were normalized to adjust location and size of 
binary image. Also its aim was to eliminate errors derived from 
camera dithering and distance changes between camera and 
human body. In this paper, silhouettes were zoomed in (or out) to 
150 x 150 pixels. Finally, the normalized silhouettes were added 
together to obtain the gait energy image. Fig. 5 shows the 
procedure of normalization. 

(2) Wavelet Decomposition of GEI: The aim of wavelet decom¬ 
position (WD) was to decompose the energy of GEI according to 
different directions. The wavelet decomposition made the GEI to 
many hierarchical images. Then we could get some energy images 
with different levels and obtain more energy features of human 
body. In view of this, we decomposed the GEI in our experiments. 
Fig. 6 shows the experimental result. The S illustrates GEI. The Ai 
(decomposition order i= 1 or 2) is approximation of S. The Hi, Vi 
and Di represent three components separately: horizontal, 
vertical and Diagonal component. According to Fig. 6, we can 
find that three components (HI, VI and Dl) have clear outline 
with energy feature. However, the other three components (H2, 
V2 and D2) have body information with limited energy. In order 
to improve computing speed, we decomposed the GEI only in one- 
layer and used the Haar wavelet in this paper. 

2.3. Feature extraction 

Firstly, the skeleton parameters were extracted by using the 
simplified model. The skeleton is an important description of 
topological relation. For the GEI with slender shape, its skeleton 
should cover important feature of human body. So we simplified 
GEI to obtain skeleton. Then skeleton parameters were extracted 
according to anatomy. The basic steps follow as 

(1) To extract skeleton of the GEI based on mathematical 
morphology. 

(2) According to body segment properties [16,17], curve fittings 
were presented for skeleton’s trunk and lower limbs as shown 
in Fig. 7. Its aim of curve fitting was not only to eliminate the 
influence from motion of upper limbs, but also to cut down 
some skeleton’s branches. 

(3) According to the four points A (xi, yi), B (x 2 , y 2 ), C (x 3 , y 3 ) 
and D (x 4 , y 4 ) as shown in Fig. 7, we computed the height of 
total skeleton: H = yi-min(y 3 ,y 4 ). Then we calculated the 
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(3) Loading package (4) Heavy coat 

Fig. 4. Infrared images with different conditions. 



Fig. 5. Pretreatment of IR image. 


following parameters: 


V /0',-y2) 2 +(X 1 -X2) 2 

1 - H 

(4) 

n J(y2-y3?+(x 2 -x 3 f 

P2 = H 

(5) 

^(y2-y 4 ) 2 +(x 2 -x 4 ) 2 

13 ~ H 

(6) 

0 _ |x 3 -x 2 | 

1 Iy3-y2i 

(7) 

a IX4-X2I 

Iy4-y2i 

(8) 


Here Pi is proportion between trunk and height of skeleton. In 
Eqs. (5)-(6), the P 2 and P 3 illustrate the relations between 
skeleton height and lower limbs. In Eqs. (7)-(8), 6\ and 0 2 
present the angular information for lower limbs. According to 
the anatomy, the height of knee joint is 0.285 H (H is the 
height of skeleton) above the ground [16,17]. We determined 
the coordinate of skeleton’s point above 0.285 H. Then these 
points were fitted and the segment AB could be obtained by 
using least square method as shown in Fig. 7. Also, we draw 
the line-segment BC and BD using the method of curve fitting. 
According to skeleton structure, we find that these skeleton 
parameters based on simplified model should contain some 
anatomical characters of human body. 


Secondly, the invariant moments were extracted by using the 
integral model. The invariant moments have the invariance of 
rotation, scaling, translation, gray change and contrast 
change. So we extracted the invariant moments based on 
integral model. The steps follow as 

(4) To decompose the gait energy image. 

(5) To compute moment parameters of approximation (S), 
horizontal component (HI), vertical component (VI) and 
diagonal component (Dl). 

(6) To compute the logarithm for moment parameters because a 
wide range for the value. 

(7) To normalize moment parameters and skeleton parameters 
respectively. 

Using this method, we could obtain 33 parameters (28 
parameters based on integral model and 5 parameters based 
on simplified model). The parameters based on simplified 
model implied anatomical characteristic. At the same time, 
the parameters based on integral model covered the beha¬ 
vioral characteristic. In this way, the gait parameters can 
include the anatomical and behavioral characteristics. 


2.4. Gait recognition based on SVM 

The traditional methods of statistical pattern recognition are 
based on the groundwork that there are sufficient samples. The 
performance can be ensured theoretically when the sample 
number tends to infinite. Theory is nice, but there are limited 
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c (x 3 ,y 3 ) D (x 4 , y 4 ) 


Fig. 7. Sketch map of key points. 


samples in gait recognition. In this case, it is difficult for some 
methods to achieve ideal results. However, support vector 
machine (SVM) is a machine learning method based on small 
sample statistics theory. This method has some advantages 
to deal with the pattern recognition involving small sample, 
non-linear, high-dimension data [18]. So we used SVM to classify 
gaits. 

The SVM is designed for binary classification. But gait 
recognition is to solve problem with multi-class classification. 


Here one of the technology difficulties is how to use SVM to 
classify multi-class gaits. In this paper, we used the method of 
‘one-to-others’ [19]. For N-class classification, the number of SVM 
classifier is N. Each classifier labels one-class from the others. The 
identification result would be decided by maximum distance of 
output value with the interface (w-x+b). Provided that m-class 
samples (Si, S 2 ,...,S m ) need to be classified. Then we designed 
classifiers: Ci, C 2 ,...,C m . Each classifier Q (i=l,2,...,m) was trained 
by the sample S t (i=l,2,...,m) which was used as positive class. 
The other samples (Sy, j =£ i) were used as negative sample for 
training classifier Q. For the positive sample, the output of SVM 
labeled +1. The system labeled -1 for the negative sample. While 
predicting the recognition performance, we input the sample T t as 
testing item. If the result labels +1 for the classifier Q, we would 
classify the sample T, to be i class. Else the result would be wrong. 

On the side, the kernel is one of most important parameters for 
SVM. Different kernel faces the different mapping space for 
testing samples. This paper presents three common kernels: 
Linear kernel, Polynomial kernel and radial basis function (RBF) 
kernel. 

(I) Linear kernel, I<(x,Xi) = x • Xy (9) 

(II) Polynomial kernal, K(x,Xi) = [(x • x* +1 )] 9 (10) 

(III) Radial basis function kernel: K(x,X;) = exp{—|x—x z | 2 /2cr 2 } (11) 

For the infrared gait video, each subject have four sequences. 
In experiments, we chose three sequences as training items and 
the other one sequence as testing item. 
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Table 1 

Comparision with ANN and SVM. 

Exp. 

Training item 

Testing item 

Parameters 

ANN 




SVM 






Linear (%) 

RBF (%) 

BP (%) 

Linear (%) 

Cubic polynomial (%) 

RBF(%) 

A 

Normal walking 
With volleyball 
With package 

Heavy coat 

Skeleton 

26 

30 

35 

30 

39 

43 

B 

Normal walking 
With volleyball 
With package 

Heavy coat 

Invariant moments 

39 

43 

48 

43 

52 

57 

C 

Normal walking 
With volleyball 
With package 

Heavy coat 

Skeleton+Invariant moments 

57 

70 

74 

65 

74 

78 


Table 2 

Results of recognition by using SVM. 


Exp. 

Training item 

Testing item 

Conventional image (%) 

IR image (%) 

1 

With volleyball, With package, Heavy coat 

Normal walking 

83 

91 

2 

Normal walking, With package, Heavy coat 

With volleyball 

74 

83 

3 

Normal walking, With volleyball, Heavy coat 

With package 

74 

87 

4 

Normal walking, With volleyball, With package 

Heavy coat 

70 

78 


3. Results and discussion 

This method was applied to infrared thermal gait database. 
There were 23 subjects (13 males and 10 females). Their age was 
from 19 years to 30 years. Each subject was asked to walk with 
four conditions: normal walking, loading package of known 
weight (5 kg), holding volleyball and wearing heavy coat. There 
are as many as four video sequences for each subject: (normal 
walking) x (loading package) x (holding volleyball) x (wearing 
heavy coat). So there are 92 (23 x 4) sequences in this infrared 
database. In order to estimate the performance of gait recognition, 
we used probability of correct recognition (PCR) as the guideline. 
While testing the recognition performance, the sample T, would be 
put into the classifiers. If the input 7) had the result i, it was 
confirmed that the output was right. If the sample T, had the result 
j (i ^ j), the output was wrong. The PCR was shown as following: 

PCR = N r /N x 100% (12) 

In Eq. (12), PCR is the probability of correct recognition. N r is the 
number of samples that was classified right and N is the number of 
total subjects. 

Table 1 gives comparative results of recognition performance for 
the single model and double model. As presented in Table 1, the PCR 
was lower than 50% when we only used the skeleton parameters. 
When the moment parameters were used to classify gaits, the PCR 
was lower than 60%. If double model (integral model+simplified 
model) was used to extract gait features, we could achieve more 
ideal results. According to the test results from Table 1, it was 
proved that this method with double model was beneficial to 
classify gaits. At the same time, this paper presents recognition 
results by using artificial neural network (ANN) with three 
networks: linear, radial basis function (RBF) and back-propagation 
(BP) as presented in Table 1. The accurate rate of the recognition 
reached 74% when using ANN with BP network as classifier. 
However, SVM could identify the class feature better. So we used 
SVM to recognize infrared gaits in this paper. In addition, The PCR 
was lower when we used SVM with linear kernel and PCR was 
improved when the polynomial kernel was used. An ideal result was 
achieved when using SVM with RBF kernel in experiment. For this 
reason, the RBF kernel was used for identification. 


Table 2 gives recognition results by using SVM with RBF kernel for 
two types of video. There are 23 subjects who were not common 
between the IR video and the conventional video. For conventional 
video, each subject was also asked to walk with four conditions: 
normal walking, loading package of known weight (5 kg), holding 
volleyball and wearing heavy coat. We detected silhouettes by using 
background subtraction. At the same time, the GEI was obtained by 
adding normalized body silhouettes together. The same methods 
were used to extract gait features for conventional video. The 
recognition rate was 70%-83%. The PCR was lower than that of IR 
video. When we trained SVM with three items (with volleyball, with 
package, heavy coat) and tested the other one for IR video, the PCR 
was 91%. The lowest recognition rate was 78% when we used the 
sequence with heavy coat as testing sample. For testing items of 
holding objects (with package and with volleyball), the PCR was 87% 
and 83%, respectively. In addition, Ref. [6] performed four 
experiments on infrared night gait dataset. The results showed that 
carrying a bag had much more impact on the recognition 
performance. The recognition rate was 60% when subjects carried a 
bag in Ref. [6]. In this paper, the recognition rate was 87% when 
subjects walked with a package. The PCR was insensitive for the 
items of holding ball and loading package. However, there was 
significant influence for the item of wearing heavy coat. As expected, 
the gait-related factors include the body silhouette. When subject 
walks normally, the silhouette will be close to his/her body structure. 
When subject walks with a heavy coat, the outline will be changed. 
This will make a larger error between silhouette and human body. So 
heavy coat affects the feature extraction based on GEI. This impact 
leads to a lower recognition rate as presented in Table 2. 


4. Conclusions 

Human body is a natural emitter of infrared ray and the 
temperature is not similar to that of surroundings. So the infrared 
image is not easy to be disturbed by complex background and 
illumination variations. Also IR imagery appears more robust for 
silhouette extraction. In view of this, we used infrared thermal 
camera to collect gait video and established an infrared thermal gait 
database. To test the recognition performance, we used double-model 
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to extract gait features and achieved 78%-91% for the probability 
of correct recognition. From our study, the infrared thermal image 
is beneficial to detect body silhouette in gait recognition. Also the 
recognition rate is insensitive for the objects (such as package and 
ball). This method is robust against noises form background, 
illumination and walking speed. However, there is significant 
influence on gait recognition when subject wears heavy coat. Our 
future work will take into account more factors, such as subject 
number, view and time. In addition, fusion of dual infrared 
information will be explored to recognize people by their gaits. 
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